Contactless Monitoring of Respiratory Rate and Breathing Absence from Head Movements Using an RGB Camera.

Contactless vital sign monitoring is more demanding for long-term, continuous, and unobtrusive measurements. Camera-based respiratory monitoring is receiving growing interest with advanced video technologies and computational power. The volume variations of the lungs for airflow changes create a periodic movement of the torso, but identifying the torso is more challenging than face detection in a video. In this paper, we present a unique approach to monitoring respiratory rate (RR) and breathing absence by leveraging head movements alone from an RGB video because respiratory motion also influences the head. Besides our novel RR estimation, an independent algorithm for breathing absence detection using signal feature extraction and machine learning techniques identifies an apnea event and improves overall RR estimation accuracy. The proposed approach was evaluated using videos from 30 healthy subjects who performed various breathing tasks. The breathing absence detector had 0.87 F1 score, 0.9 sensitivity, and 0.85 specificity. The accuracy of spontaneous breathing rate estimation increased from 2.46 to 1.91 bpm MAE and 3.54 to 2.7 bpm RMSE when combining the breathing absence result with the estimated RR.Clinical relevance- Our contactless respiratory monitoring can utilize a consumer RGB camera to offer a significant benefit in continuous monitoring of neonatal monitoring, sleep monitoring, telemedicine or telehealth, home fitness with mild physical movement, and emotion detection in the clinic and remote locations.

Enhanced Contactless Heart Rate Monitoring Using Camera with Motion Artifact Removal During Physical Activities.

Contactless monitoring of heart rate (HR) can improve passive and continuous tracking of cardiovascular activities and overall people's health. Remote photoplethysmography (rPPG) using a camera eliminates the need for a wearable device. rPPG-based HR has shown promising results to be accurate and comparable to conventional methods such as contact PPG. Most experiments use stationary subjects while motion is known to affect the accuracy of remote PPG. In this paper, a novel methodology is introduced to enhance the accuracy and reliability of HR monitoring based on rPPG in the presence of physical activities like Yoga. This method quickly and accurately tracks HR and analyzes head motion to exclude unreliable data within short windows of rPPG signals. The method was tested with smartphone video data collected from 60 subjects when they are doing activities with varying levels of movement. Results show that our method without motion removal improves the accuracy of the HR readings by 0.7 bpm, reaching 3.57 bpm on average for a 30-sec-window. The accuracy is further improved by another 1.3 bpm after removing the motion artifacts, and reaches 2.29 bpm.Clinical relevance- The enhancement of HR readings from shorter rPPG signal with motion tolerance during physical activities can ultimately help with a more reliable HR tracking of people in uncontrolled settings like home which is a critical step towards remote health-care or wellness tracking.

MM-HAR: Multi-Modal Human Activity Recognition Using Consumer Smartwatch and Earbuds.

Human Activity Recognition (HAR) is one of the important applications of digital health that helps to track fitness or to avoid sedentary behavior by monitoring daily activities. Due to the growing popularity of consumer wearable devices, smartwatches, and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 44 subjects collected from both in-lab and in-home environments, we demonstrate the limitations of using a single modality as well as the importance of a multi-modal approach. Moreover, we also train and evaluate the performance of five different machine learning classifiers for various combinations of devices such as buds only, watch only, and both. We believe the detailed analyses presented in this paper may serve as a benchmark for the research community to explore and build upon in the future.

Power Optimized Smartwatch-Earbuds Multimodal System for Monitoring Activities of Daily Living.

Activities of daily living is an important entity to monitor for promoting healthy lifestyle for chronic disease patients, children and the healthy population. This paper presents a smartwatch and earbuds inertial sensors based multi-modal power efficient end-to-end mobile system for continuous, passive and accurate detection of broad daily activity classes. We collected various posture, stationary and moving activity data from 40 diverse subjects using earbuds and smartwatch and develop the novel power optimized end-to-end operational system consisting of i) optimized device sampling rates and Bluetooth packet transfer rates, ii) data buffering mechanism, iii) background services, and iv) optimized model size, and demonstrating 93% macro recall score in detecting various activities. Our power optimized solution uses 80%, 40% and 33.33% less battery power for the smartphone, smartwatch, and earbuds respectively, compared to a power agnostic system with an estimated continuous no-charging run time of 50 hours, 16.67 hours, and 25 hours for the smartphone, smartwatch, and earbuds respectively.Clinical relevance- The end-to-end power optimized activity detection system presented in this paper will assist practicing clinicians toward treatment of various chronic disease patients (e.g. diabetes, hypertension, heart disease and obesity) by long-term, continuous monitoring of their lifestyle and sedentary behavior.

RRDetection: Respiration Rate Estimation Using Earbuds During Physical Activities.

In modern times, earbuds have become both popular and essential accessories for people to use with a wide range of devices in their everyday lives. Moreover, the respiration rate is a crucial vital sign that is sensitive to various pathological conditions. Many earbuds now come equipped with multiple sensing capabilities, including inertial and acoustic sensors. These sensors can be used by researchers to passively monitor users' vital signs, such as respiration rates. While current earbud-based breath rate estimation algorithms mostly focus on resting conditions, recent studies have demonstrated that respiration rates during physical activities can predict cardio-respiratory fitness for healthy individuals and pulmonary conditions for respiratory patients. To address this gap, we propose a novel algorithm called RRDetection that leverages the motion sensors in ordinary earbuds to detect respiration rates during light to moderate physical activities.

Estimating SpO2 with Deep Oxygen Desaturations from Facial Video Under Various Lighting Conditions: A Feasibility Study.

This paper presents a feasibility study to collect data, process signals, and validate accuracy of peripheral oxygen saturation (SpO2) estimation from facial video in various lighting conditions. We collected facial videos using RGB camera, without auto-tuning, from subjects when they were breathing through a mouth tube with their nose clipped. The videos were record under four lighting conditions: warm color temperature and normal brightness, neutral color temperature and normal brightness, cool color temperature and normal brightness, neutral color temperature and dim brightness. The air inhaled by the subjects was manually controlled to gradually induce hypoxemia and lower subjects' SpO2 to as low as 81%. We first extracted the remote photoplethysmogram (rPPG) signals from the videos. We applied the principle of pulse oximetry and extracted the ratio of ratios (RoR) for two color combinations: Red/Blue and Red/Green. Next, we assessed SpO2 estimation accuracy against the ground truth, a Transfer Standard Pulse Oximeter. We have achieved an RMSE of 1.93% and a PCC of 0.97 under the warm color temperature and normal brightness lighting condition using leave-one-subject-out cross validation between two subjects. The results have demonstrated the feasibility to estimate SpO2 remotely and accurately using consumer level RGB camera with suitable camera configuration and lighting condition.Clinical Relevance- This work demonstrates that SpO2 can be estimated accurately using an RGB camera without auto-tuning and under warm color temperature, enabling continuous SpO2 monitoring applications that require noncontact sensing.

Utilizing Deep Learning on Limited Mobile Speech Recordings for Detection of Obstructive Pulmonary Disease.

Passive assessment of obstructive pulmonary disease has gained substantial interest over the past few years in the mobile and wearable computing communities. One of the promising approaches is speech-based pulmonary assessment wherein spontaneous or scripted speech is used to evaluate an individual's pulmonary condition. Recent approaches in this regard heavily rely on accurate speech activity segmentation and specific, hand-crafted features. In this paper, we present an end-to-end deep learning approach for detecting obstructive pulmonary disease. We leveraged transfer learning using a network pre-trained for a different audio-based task, and employed our own additional shallow network on top as a binary classifier to indicate if a given speech recording belongs to an asthma or COPD patient. The additional network was a fully connected neural net with 2 hidden layers, and this was evaluated on two real-world datasets. We demonstrated that the system can identify subjects with obtructive pulmonary disease using their speech with 88.3 % precision, 88.8 % recall and 88.3% F-1 score using 10-fold cross-validation. The system showed improved performance in identifying the most severely affected subgroup of patients in the dataset, with an average 93.6 % accuracy.

Deep Multivariate Domain Translation for Device Invariant Pulmonary Patient Identification from Cough and Speech Sounds.

Pulmonary audio sensing from cough and speech sounds in commodity mobile and wearable devices is increasingly used for remote pulmonary patient monitoring, home healthcare, and automated disease analysis. Patient identification is important for such applications to ensure system accuracy and integrity, and thus avoiding errors and misdiagnosis. Widespread usage and deployment of such patient identification models across various devices are challenging due to domain shift of acoustic features because of device heterogeneity. Because of this phenomenon, a patient identification model developed using audio data collected with one type of device is not usable when deployed in another type of device, which is a concern for model portability and general usability. This paper presents a framework utilizing a multivariate deep neural network regressor as a feature translator between source device and target device domains to reduce the effect of domain shift for better model portability. Extensive and empirical experiments of our translation framework consisting of two different human sound (speech and cough) based pulmonary patient identification tasks using audio data collected from 91 real patients demonstrate that it can recover up to 64.8% of lost accuracy due to domain shift across two common and widely used mobile and wearable devices: smartphone and smartwatch. Clinical Relevance- The methods presented in this paper will enable efficient and easy portability of pulmonary patient identification models from cough and speech across various mobile and wearable devices used by a patient.

Detecting Physiological Responses Using Multimodal Earbud Sensors.

Continuous stress exposure negatively impacts mental and physical well-being. Physiological arousal due to stress affects heartbeat frequency, changes breathing pattern and peripheral temperature, among several other bodily responses. Traditionally stress detection is performed by collecting signals such as electrocardiogram (ECG), respiration, and skin conductance response using uncomfortable sensors such as a chestband. In this study, we use earbuds that passively measure photoplethysmography (PPG), core body temperature, and inertial measurements. We have conducted a lab study exposing 18 participants to an evaluated speech task and additional tasks aimed at increasing stress or promoting relaxation. We simultaneously collected PPG, ECG, impedance cardiography (ICG), and blood pressure using laboratory grade equipment as reference measurements. We show that the earbud PPG sensor can reliably capture heart rate and heart rate variability. We further show that earbud signals can be used to classify the physiological responses associated with stress with 91.30% recall, 80.52% precision, and 85.12% F1-score using a random forest classifier with leave-one-subject-out cross-validation. The accuracy can further be improved through multi-modal sensing. These findings demonstrate the feasibility of using earbuds for passively monitoring users' physiological responses.

Remote Photoplethysmography and Heart Rate Estimation by Dynamic Region of Interest Tracking.

Remote photoplethysmography (PPG) estimates vital signs by measuring changes in the reflected light from the human skin. Compared to traditional PPG techniques, remote PPG enables contactless measurement at a reduced cost. In this paper, we propose a novel method to extract remote PPG signals and heart rate from videos. We propose an algorithm to dynamically track regions of interest (ROIs) and combine the signals from all ROIs based on signal qualities. To maintain a stable frame rate and accuracy, we propose a dynamic down-sampling approach, which makes our system robust to the different video resolutions and user-camera distances. We also propose the strategy of adaptive measurement time to estimate HR, which can achieve comparable accuracy in HR estimation while reducing the average measurement time. To test the accuracy of the proposed system, we have collected data from 30 subjects with facial masks. Experimental results show that the proposed system can achieve 3.0 bpm mean absolute error in HR estimation.

Motion- Based Respiratory Rate Estimation with Motion Artifact Removal Using Video of Face and Upper Body.

Respiratory rate (RR) is a significant indicator of health conditions. Remote contactless measurement of RR is gaining popularity with recent respiratory tract infection awareness. Among various methods of contactless RR measurement, a video of an individual can be used to obtain an instantaneous RR. In this paper, we introduce an RR estimation based on the subtle motion of the head or upper chest captured on an RGB camera. Motion-based respiratory monitoring allows us to acquire RR from individuals with partial face coverings, such as glasses or a face mask. However, motion-based RR estimation is vulnerable to the subject's voluntary movement. In this work, adaptive selection between face and chest regions plus a motion artifact removal technique enables us to obtain a much cleaner respiratory signal from the video recordings. The average mean absolute error (MAE) for controlled and natural breathing is 1.95 BPM using head motion only and 1.28 BPM using chest motion only. Our results demonstrate the possibility of continuous monitoring of breathing rate in real-time with any personal device equipped with an RGB camera, such as a laptop or a smartphone.

Atrial Fibrillation Detection and Atrial Fibrillation Burden Estimation via Wearables.

Atrial Fibrillation (AF) is an important cardiac rhythm disorder, which if left untreated can lead to serious complications such as a stroke. AF can remain asymptomatic, and it can progressively worsen over time; it is thus a disorder that would benefit from detection and continuous monitoring with a wearable sensor. We develop an AF detection algorithm, deploy it on a smartwatch, and prospectively and comprehensively validate its performance on a real-world population that included patients diagnosed with AF. The algorithm showed a sensitivity of 87.8% and a specificity of 97.4% over every 5-minute segment of PPG evaluated. Furthermore, we introduce novel algorithm blocks and system designs to increase the time of coverage and monitor for AF even during periods of motion noise and other artifacts that would be encountered in daily-living scenarios. An average of 67.8% of the entire duration the patients wore the smartwatch produced a valid decision. Finally, we present the ability of our algorithm to function throughout the day and estimate the AF burden, a first-of-this-kind measure using a wearable sensor, showing 98% correlation with the ground truth and an average error of 6.2%.

RRMonitor: A Resource-Aware End-to-End System for Continuous Monitoring of Respiration Rate Using Earbuds.

Respiration rate is considered as a critical vital sign, and daily monitoring of respiration rate could provide helpful information about any acute condition in the human body. While researchers have been exploring mobile devices for respiration rate monitoring, passive and continuous monitoring is still not feasible due to many usability challenges (e.g., active participation) in existing approaches. This paper presents an end-to-end system called RRMonitor that leverages the movement sensors from commodity earbuds to continuously monitor the respiration rate in near real-time. While developing the systems, we extensively explored some key parameters, algorithms, and approaches from existing literature that are better suited for continuous and passive respiration rate monitoring. RRMonitor can passively track the respiration rate with a mean absolute error as low as 1.64 cycles per minute without requiring active participation from the user.

Device Invariant Deep Neural Networks for Pulmonary Audio Event Detection Across Mobile and Wearable Devices.

Mobile and wearable devices are being increasingly used for developing audio based machine learning models to infer pulmonary health, exacerbation and activity. A major challenge to widespread usage and deployment of such pulmonary health monitoring audio models is to maintain accuracy and robustness across a variety of commodity devices, due to the effect of device heterogeneity. Because of this phenomenon, pulmonary audio models developed with data from one type of device perform poorly when deployed on another type of device. In this work, we propose a framework incorporating feature normalization across individual frequency bins and combining task specific deep neural networks for model invariance across devices for pulmonary event detection. Our empirical and extensive experiments with data from 131 real pulmonary patients and healthy controls show that our framework can recover up to 163.6% of the accuracy lost due to device heterogeneity for four different pulmonary classification tasks across two broad classification scenarios with two common mobile and wearable devices: smartphone and smartwatch.Clinical relevance- The methods presented in this paper will enable efficient and easy portability of clinician recommended pulmonary audio event detection and analytic models across various mobile and wearable devices used by a patient.

Real-Time Limb Motion Tracking with a Single IMU Sensor for Physical Therapy Exercises.

Limb exercises are common in physical therapy to improve range of motion (RoM), strength, and flexibility of the arm/leg. To improve therapy outcomes and reduce cost, motion tracking systems have been used to monitor the user's movements when performing the exercises and provide guidance. Traditional motion tracking systems are based on either cameras or inertial measurement unit (IMU) sensors. Camera-based systems face problems caused by occlusion and lighting. Traditional IMU-based systems require at least two IMU sensors to track the motion of the entire limb, which is not convenient for use. In this paper, we propose a novel limb motion tracking system that uses a single 9-axis IMU sensor that is worn on the distal end joint of the limb (i.e., wrist for the arm or ankle for the leg). Limb motion tracking using a single IMU sensor is a challenging problem because 1) the noisy IMU data will cause drift problem when estimating position from the acceleration data, 2) the single IMU sensor measures the motion of only one joint but the limb motion consists of motion from multiple joints. To solve these problems, we propose a recurrent neural network (RNN) model to estimate the 3D positions of the distal end joint as well as the other joints of the limb (e.g., elbow or knee) from the noisy IMU data in real time. Our proposed approach achieves high accuracy with a median error of 7.2/7.1 cm for the wrist/elbow joint in leave-one-subject-out cross validation when tracking the arm motion, outperforming the state-of-the-art approach by more than 10%. In addition, the proposed model is lightweight, enabling real-time applications on mobile devices.Clinical relevance- This work has great potential to improve limb exercises monitoring and RoM measurement in home-based physical therapy. It is also cost effective and can be made available widely for immediate application.

SpeechSpiro: Lung Function Assessment from Speech Pattern as an Alternative to Spirometry for Mobile Health Tracking.

Respiratory illnesses are common in the United States and globally; people deal with these illnesses in various forms, such as asthma, chronic obstructive pulmonary diseases, or infectious respiratory diseases (e.g., coronavirus). The lung function of subjects affected by these illnesses degrades due to infection or inflammation in their respiratory airways. Typically, lung function is assessed using in-clinic medical equipment, and quite recently, via portable spirometry devices. Research has shown that the obstruction and restriction in the respiratory airways affect individuals' voice characteristics. Hence, audio features could play a role in predicting the lung function and severity of the obstruction. In this paper, we go beyond well-known voice audio features and create a hybrid deep learning model using CNN-LSTM to discover spatiotemporal patterns in speech and predict the lung function parameters with accuracy comparable to conventional devices. We validate the performance and generalizability of our method using the data collected from 201 subjects enrolled in two studies internally and in collaboration with a pulmonary hospital. SpeechSpiro measures lung function parameters (e.g., forced vital capacity) with a mean normalized RMSE of 12% and R2 score of up to 76% using 60-second phone audio recordings of individuals reading a passage.Clinical relevance - Speech-based spirometry has the potential to eliminate the need for an additional device to carry out the lung function assessment outside clinical settings; hence, it can enable continuous and mobile track of the individual's condition, healthy or with a respiratory illness, using a smartphone.

A Novel Multi-Centroid Template Matching Algorithm and Its Application to Cough Detection.

Cough is a major symptom of respiratory-related diseases. There exists a tremendous amount of work in detecting coughs from audio but there has been no effort to identify coughs from solely inertial measurement unit (IMU). Coughing causes motion across the whole body and especially on the neck and head. Therefore, head motion data during coughing captured by a head-worn IMU sensor could be leveraged to detect coughs using a template matching algorithm. In time series template matching problems, K-Nearest Neighbors (KNN) combined with elastic distance measurement (esp. Dynamic Time Warping (DTW)) achieves outstanding performance. However, it is often regarded as prohibitively time-consuming. Nearest Centroid Classifier is thereafter proposed. But the accuracy is comprised of only one centroid obtained for each class. Centroid-based Classifier performs clustering and averaging for each cluster, but requires manually setting the number of clusters. We propose a novel self-tuning multi-centroid template-matching algorithm, which can automatically adjust the number of clusters to balance accuracy and inference time. Through experiments conducted on synthetic datasets and a real-world earbud-based cough dataset, we demonstrate the superiority of our proposed algorithm and present the result of cough detection with a single accelerometer sensor on the earbuds platform.Clinical relevance- Coughing is a ubiquitous symptom of pulmonary disease, especially for patients with COPD and asthma. This work explores the possibility and and presents the result of cough detection using an IMU sensor embedded in earables.

Validation of an algorithm for continuous monitoring of atrial fibrillation using a consumer smartwatch.

BACKGROUND: Consumer devices with broad reach may be useful in screening for atrial fibrillation (AF) in appropriate populations. However, currently no consumer devices are capable of continuous monitoring for AF. OBJECTIVE: The purpose of this study was to estimate the sensitivity and specificity of a smartwatch algorithm for continuous detection of AF from sinus rhythm in a free-living setting. METHODS: We studied a commercially available smartwatch with photoplethysmography (W-PPG) and electrocardiogram (W-ECG) capabilities. We validated a novel W-PPG algorithm combined with a W-ECG algorithm in a free-living setting, and compared the results to those of a 28-day continuous ECG patch (P-ECG). RESULTS: A total of 204 participants completed the free-living study, recording 81,944 hours with both P-ECG and smartwatch measurements. We found sensitivity of 87.8% (95% confidence interval [CI] 83.6%-91.0%) and specificity of 97.4% (95% CI 97.1%-97.7%) for the W-PPG algorithm (every 5-minute classification); sensitivity of 98.9% (95% CI 98.1%-99.4%) and specificity of 99.3% (95% CI 99.1%-99.5%) for the W-ECG algorithm; and sensitivity of 96.9% (95% CI 93.7%-98.5%) and specificity of 99.3% (95% CI 98.4%-99.7%) for W-PPG triggered W-ECG with a single W-ECG required for confirmation of AF. We found a very strong correlation of W-PPG in quantifying AF burden compared to P-ECG (r = 0.98). CONCLUSION: Our findings demonstrate that a novel algorithm using a commercially available smartwatch can continuously detect AF with excellent performance and that confirmation with W-ECG further enhances specificity. In addition, our W-PPG algorithm can estimate AF burden. Further research is needed to determine whether this algorithm is useful in screening for AF in select at-risk patients.

A Comprehensive Approach for Classification of the Cough Type.

Identifying the presence of sputum in the lung is essential in detection of diseases such as lung infection, pneumonia and cancer. Cough type classification (dry/wet) is an effective way of examining presence of lung sputum. This is traditionally done through physical exam in a clinical visit which is subjective and inaccurate. This work proposes an objective approach relying on the acoustic features of the cough sound. A total number of 5971 coughs (5242 dry and 729 wet) were collected from 131 subjects using Smartphone. The data was reviewed and annotated by a novel multi-layer labeling platform. The annotation kappa inter-rater agreement score is measured to be 0.81 and 0.37 for 1st and 2nd layer respectively. Sensitivity and specificity values of 88% and 86% are measured for classification between wet and dry coughs (highest across the literature).

CoughGAN: Generating Synthetic Coughs that Improve Respiratory Disease Classification.

Despite the prevalence of respiratory diseases, their diagnosis by clinicians is challenging. Accurately assessing airway sounds requires extensive clinical training and equipment that may not be easily available. Current methods that automate this diagnosis are hindered by their use of features that require pulmonary function tests. We leverage the audio characteristics of coughs to create classifiers that can distinguish common respiratory diseases in adults. Moreover, we build on recent advances in generative adversarial networks to augment our dataset with cleverly engineered synthetic cough samples for each class of major respiratory disease, to balance and increase our dataset size. We experimented on cough samples collected with a smartphone from 45 subjects in a clinic. Our CoughGAN-improved Support Vector Machine and Random Forest models show up to 76% test accuracy and 83% F1 score in classifying subjects' conditions between healthy and three major respiratory diseases. Adding our synthetic coughs improves the performance we can obtain from a relatively small unbalanced healthcare dataset by boosting the accuracy over 30%. Our data augmentation reduces overfitting and discourages the prediction of a single, dominant class. These results highlight the feasibility of automatic, cough-based respiratory disease diagnosis using smartphones or wearables in the wild.